Post capture imagery processing and deployment systems

ABSTRACT

A post capture imagery processing system is provided. The system is for use with aerial imagery and includes a server having a processor and a memory and a software application providing instruction to the server to process the captured aerial imagery, such as spherical imagery. The server further includes instructions to geo-rectify the spherical imagery. The geo rectifying of the spherical imagery may include one of use of a third party GIS map to associate corresponding data with the spherical imagery in order to produce a geo-referenced spherical image, or calculate the geo-references by a software application performing particular operations on the server.

CROSS REFERENCE TO RELATED APPLICATION[S]

This application claims priority to U.S. Provisional Patent Applicationentitled “INTERFACE FOR DEPLOYMENT OF AERIAL CAPTURED IMAGERY,” Ser. No.62/126,916, filed Mar. 2, 2015, and claims priority to U.S. ProvisionalPatent Application entitled “POST CAPTURE IMAGERY PROCESSING SYSTEM,”Ser. No. 62/126,919, filed Mar. 2, 2015, the disclosures of which arehereby incorporated entirely herein by reference.

BACKGROUND OF THE INVENTION

Technical Field

This invention relates generally to an image processing system and moreparticularly to a post capture imagery processing system and adeployment interface of aerial captured spherical imagery.

State of the Art

Aerial photography is useful for many different purposes. For example,aerial photography is used for cartography, land-use planning,archaeology, movie production, environmental studies, surveillance,commercial advertising, conveyancing, artistic projects, and many otherpurposes. The imagery that is captured is often used with a geographicinformation system (“GIS”) to provide additional information.

Various business and entities are utilizing aerial imagery in order tomarket and provide better service to customers and the like by using GISmaps and the like. However, a system of effectively processing aerialspherical imagery after capture of the aerial imagery is lacking.

There is a lack of an interface of deploying aerial spherical imagery inorder to provide a useful image that can be accessed by variousindustries and people.

Accordingly, there is a need for a post capture imagery processingsystem and an interface for deployment of aerial spherical imagery.

SUMMARY OF THE INVENTION

The present invention relates to a post capture imagery processingsystem for aerial spherical imagery. The system produces a sphericalimage that is geo-referenced or in other words, the system geo-rectifiesthe spherical imagery. A common thought when saying this is that thesphere is in a known location when it was captured. The systemaccomplishes this by not only having a known location when capturing theimage, but further rectifying the image such that the system associatesthe latitude and longitude of each pixel in the spherical image.

An embodiment include a post capture imagery processing system for usewith aerial spherical imagery, the system comprising: a servercomprising a memory, a database stored in the memory, and a processor,wherein the database contains data for a spherical image and data for areference map; and a computer coupled to the server wherein a userinterfaces with the server through the computer, wherein the server isprogrammed to: automatically create a non-linear warp from a referencegeodetic coordinate system to a pan and tilt coordinates in anequirectangular image; display the equirectangular image on a display ofthe computer; automatically retrieving a georectified location of apoint in the equirectangular image in response to receiving a signalfrom the computer.

To automatically create the non-linear warp the server may be furtherprogrammed to automatically perform a registration process, whereinregistration process is a manual registration process using a thin platespline warping process or wherein the registration process is anautomated registration process.

To automatically create the non-linear warp, the server is furtherprogrammed to convert a non-warped geographic map to a WGS84 Geodediccoordinate system; to convert from the WGS84 Geodedic coordinate systemto Earth-Centered-Earth-Fixed coordinate system; to convert from theEarth-Centered-Earth-Fixed coordinate system to an East-North-Up localcoordinate system; and to convert from the East-North-Up localcoordinate system to a spherical image, wherein the conversiongeorectifies the spherical image.

Another embodiment includes a method of using a post capture imageryprocessing system to georectify an aerial spherical image, the methodcomprising: storing data for a spherical image and data for a referencemap in a database of a server comprising a memory and a processor,wherein the database is stored in the memory; coupling the server to acomputer, wherein a user interfaces with the server through thecomputer; automatically creating at the server a non-linear warp from areference geodetic coordinate system to a pan and tilt coordinates in anequirectangular image; display at the computer from the server theequirectangular image; and automatically retrieving a georectifiedlocation of a point in the equirectangular image by the server inresponse to receiving a signal from the computer.

The method may further comprise performing a registration process at theserver, wherein performing the registration process comprises performinga manual registration process using a thin plate spline warping process,or wherein performing the registration process comprises performing anautomated registration process.

The method, wherein automatically creating the non-linear warp furthercomprises converting a non-warped geographic map to a WGS84 Geodediccoordinate system at the server; converting from the WGS84 Geodediccoordinate system to Earth-Centered-Earth-Fixed coordinate system at theserver; converting from the Earth-Centered-Earth-Fixed coordinate systemto an East-North-Up local coordinate system at the server; andconverting from the East-North-Up local coordinate system to a sphericalimage at the server, wherein the conversion georectifies the sphericalimage.

Yet another embodiment includes a system providing deployment interfacefor utilizing georectified aerial spherical imagery, the systemcomprising: a server comprising a memory, a database stored in thememory, and a processor, wherein the database contains data for ageorectified spherical image and data for a reference map; and acomputer coupled to the server wherein a user interfaces with the serverthrough the computer, wherein the server is programmed to: automaticallysend data for a georectified spherical image from the server to thecomputer to display of the computer the georectified spherical image;receive a signal from the computer at the server in response to the userselecting a point in the georectified spherical image using thecomputer; automatically perform a reverse warp process to determine atrue geo location of the selected point; and automatically displaying atthe computer using the deployment interface the true geo location of thepoint on a reference map in response to the server sending data of thereference map and the true geo location of the point to the computer.

To automatically perform a reverse warp, the server is furtherprogrammed to create a vector by connecting a center of the georectifiedspherical image with the point, wherein the vector is represented in theEast-North-Up coordinate system; to transform the selected point toEarth-Centered-Earth-Fixed coordinate system; to use the vector andproject it with a WGS84 ellipsoid representing earth to obtain thespherical image point; to convert from the East-North-Up localcoordinate system to a spherical image, wherein the conversiongeorectifies the spherical image; to convert from the spherical image tothe East-North-Up local coordinate system; to convert from theEast-North-Up local coordinate system to the Earth-Centered-Earth-Fixedcoordinate system; to convert from the Earth-Centered-Earth-Fixedcoordinate system to the WGS84 Geodedic coordinate system to; and toconvert the WGS84 Geodedic coordinate system to a non-warped geographicmap.

The foregoing and other features and advantages of the present inventionwill be apparent from the following more detailed description of theparticular embodiments of the invention, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the present invention may be derived byreferring to the detailed description and claims when considered inconnection with the Figures, wherein like reference numbers refer tosimilar items throughout the Figures, and:

FIG. 1 is a flow chart depicting a method of forward warping inaccordance with embodiments of the invention;

FIG. 2a is diagrammatic view of error caused during capture of an aerialspherical image, in accordance with embodiments of the invention;

FIG. 2b is a diagrammatic view of utilizing a first point for errorcorrection;

FIG. 2c is a diagrammatic view of utilizing a second point for errorcorrection, in accordance with embodiments of the invention;

FIG. 3 is a view of a screen of the system utilized for georectifying aspherical image, in accordance with embodiments of the invention;

FIG. 4a is a view of a captured spherical image, in accordance withembodiments of the invention;

FIG. 4b is a view of a matching sky image aligned with the sphericalimage, in accordance with embodiments of the invention;

FIG. 4c is a view depicting a blended matching sky image with thespherical image, in accordance with embodiments of the invention;

FIG. 5 is a diagrammatic view of a reverse warp process, in accordancewith embodiments of the invention;

FIG. 6 is view depicting relief displacement, in accordance withembodiments of the invention;

FIGS. 7a-7d are views of an iterative process for correcting reliefdisplacement, in accordance with embodiments of the invention; and

FIG. 8 is view of a deployment interface displaying a georectifiedspherical image, in accordance with embodiments of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

As discussed above, embodiments of the present invention relate to apost capture imagery processing system for aerial imagery. The systemproduces a spherical image that is geo-referenced or in other words, thesystem geo-rectifies the spherical imagery. Embodiments also relate to adeployment interface of aerial captured imagery.

Spherical imagery may be uploaded to a post capture imagery processingsystem by storing the spherical imagery on a server having a processorand a memory and a software application providing instruction to theserver to process aerial imagery, such as spherical imagery. The systemmay include a computer having a processor and a memory coupled to theserver, wherein a user can access the server through the computer inorder to process the imagery. The processing of the imagery may includeprocessing it in two different ways. Further, according to an embodimentof the invention, an interface for deployment of aerial captured imageryincludes a server having a processor and a memory and a softwareapplication providing instruction to the server to display sphericalimagery, to a user on computer coupled to the server, such as through anetwork connection, such as through the Internet. The interface mayinclude the use a lookup table using spherical bilinear interpolation togo from geodetic position to panorama coordinates directly, and usesanother lookup table to transform panorama coordinate to the approximateclosest geodetic position in order to compute on the fly the GISinformation associated with a selected or viewed portion of the image.

The interface may be deployed from a server having a processor and amemory and a software application providing instruction to the server todisplay aerial imagery, such as spherical imagery, to a user through anetwork connection, such as through the Internet. For example the systemmay provide the map as a web site to the user that is displayed on aremote computing device accessible by the user in order to view thespherical imagery.

The post capture imagery processing system operates to create anonlinear warp from a reference geodetic (latitude, longitude)coordinate system to the pan and tilt coordinates in an equirectangularimage. The post capture imagery processing system also operates toperform inverse transformation from (pan, tilt) of an equirectangularimage to the reference geodetic coordinate system. This is complicatedby the fact that there are nonlinear distortions between the observedlocations of features in the panorama and the reference image. Thesedistortions may be caused by several factors, including, but not limitedto the panorama stitching process in which several partially overlappingphotographs are superimposed and warped into a 360 degree by 180 degreeequirectangular image. Other sources of inaccuracy include errors in therecorded image geo-position information, tipping or tilting of thecameras, barrel distortion, or relief displacement from variations inelevations of features on the ground that causes elevated features toappear to lean away from the camera.

The post capture imagery processing system allows a user to visualizethe correspondence between reference map features and a panorama image,and then to edit a nonlinear warping function by placing matched pairsof points on the reference map and the panorama. The post captureimagery processing system works in conjunction with the deploymentinterface, which is a web based application for displayingaddresses/locations searched using a third party program, such as, butnot limited to Google Places API, and utilizes the mapping from pan andtilt coordinates to accurately mark the location of features in thespherical image.

There are many variables that come into play that complicate theprojection from geodetic coordinates to the pan-tilt coordinates of anequirectangular image. First, the reference map images are in a geodeticcoordinate system and the post capture image is a spherical imagecentered at a camera's position and with a pole directly below thecamera. Because of this, the projections and going between the geodeticcoordinates and the pan-tilt coordinates is not a linear transformation,but a non-linear transformation.

In linear projections, lines are mapped to lines which is not the casewhen transforming geometry between a spherical image to comparably flatimage and vice versa. The post capture imagery processing systemoperates to map straight lines in flat images to curves or warped linesin the spherical image and vice versa.

Areas around poles, like that of the Earth, create special situationsthat have given mathematicians headaches for a long time. The problem iscalled a singularity wherein all meridians collapse to a point so thearea near the pole becomes computationally unstable creating “jumps”i.e. when moving a small epsilon on the sphere creates a long jump in aflat image. In forming equirectangular images, the images are taken withthe camera pointing down at earth while shooting the images. The pointat nadir i.e. directly below the camera, is our local “pole” point onthe sphere. That is why there is huge distortion near the bottom thirdof an equirectangular image. Accordingly, there is a pole issue andhence singularity/instability when approaching the area directly belowthe camera or when you view the image on a display interface at thecenter of the spherical image. Embodiments of the present inventionaddress these tricky issues.

Points in a spherical image do not represent true geographic locationsdue to errors that exist. Because of this, there is a need to create awarp model that mathematically take points in a spherical image andrepresent them as close as possible to true geo points. As shown in FIG.1, the system include a GIS feature 12, for example, and withoutlimitation, a house, a building, and the like, represented by Pano pointPi 14 in the spherical image and Geo point Gi 16 in a true georectifiedmap. Each point Pi, Gi is a collection of lambda, theta andh—representing latitude, longitude and height respectively. In a 2Dsystem h=0, but when using DEM in 3D systems h is non-zero. In order togeorectify the spherical image, it must go through a registrationprocess, wherein the registration process may be manual or automated.

A manual registration process may be accomplished using Thin PlateSpline (“TPS”) warping 18. Data in the form of a collection of ‘N’points P1 . . . Pn and corresponding georectified points G1 . . . Gn isgathered, where N depends on distortion and errors in the sphericalimage. A user of the system may then rely on his/her skill and knowledgeto pick N such points and manually register the points. The systemprovides a user interface that interactively prompts and displaysinformation to the user in order to guide the user through the manualregistration process.

An auto registration process of the spherical image may also be providedby the system. The result of the registration process is referred to asa Warp Wf (forward warp) that converts any point in the spherical imageto a point {circumflex over (P)}i in ECEF 20. These points are thenconverted to an ENU coordinate system 22 used by the deploymentinterface 24.

A plane that is utilized for obtaining the images used to form thespherical image has a GPS system which is accurate for at least the X, Yposition of the plane. In some planes the altitude (Z position) haserror. However, the major error is due to the plane being buffetedaround while flying as well as the tilt, pitch and yaw caused rotationalerrors. An error of just a few degrees at the plane level can result ina large error for a point on earth from its true location. Referring toFIGS. 2a-2c , the error is represented.

FIG. 2a depicts the pitch or tilt error. In addition to this there maybe a yaw error associated with the plane. The position of the plane isrepresented as GPS=ϕ, λ, h, wherein ϕ=GPS latitude, λ=GPS longitude, andh=GPS altitude. The system automatically performs the followingcalculations and with reference to FIGS. 2b-2c in order to correct theerror:

-   -   X1R, Y1R=longitude, latitude on a reference map/image for a        first point    -   X1P, Y1P=longitude, latitude on a spherical image for a first        point    -   X2R, Y2R=longitude, latitude on a reference map/image for a        second point    -   X2P, Y2P=longitude, latitude on a spherical image for a second        point    -   h′=GPS altitude, adjusted    -   θz, θx, θy=yaw, tilt, pitch    -   ΔY1=Y1R−Y1P    -   ΔY1=Y1R−Y1P    -   ΔY2=Y2R−Y2P    -   ΔX2=X2R−X2P    -   rR=√{square root over ((X2R−X1R)²+(Y2R−Y1R)²)}    -   rP=√{square root over ((X2P−X1P)²+(Y2P−Y1P)²)}    -   # Calculations for first point

${\left. \begin{matrix}{{\Delta\; Y\; 1} = {h\mspace{14mu}\tan\mspace{14mu}\theta\; y}} \\{{\Delta\; Y\;{1/h}} = {\tan\mspace{14mu}\theta\; y}} \\{{a\;{\tan\left( {\Delta\; Y\;{1/h}} \right)}} = {\theta\; y}}\end{matrix} \right\}\mspace{14mu}{if}\mspace{14mu} h} = h^{\prime}$

-   -   # Calculations for second point        -   s=rR/rP        -   h′=hs        -   θz=a tan [(sY2P−sY1P−Y2R+Y1R)/(sX2P−sX1P−X2R+X1R)]        -   θy=a tan [(sY1P−Y1R)/h′]        -   θx=a tan [(sX1P−X1R)/h′]

There are two steps in the process for auto registration. These stepsinclude 1) finding matching reference points automatically, and 2)correcting for GPS errors.

When finding matching reference points automatically, there are one ofmany algorithms available to perform this function and the system mayutilize any algorithm that accurately matched reference pointsautomatically.

Correcting for GPS errors includes the system operating the softwareapplication in order to automatically perform the error correction usingthe equations described above and with reference to FIGS. 2a-2c . Thisstep requires two known points (including height if available usingDEM). Using the first point, the system is able to determine the tiltand pitch alignments. Using the second matching point a completesolution can be derived including the discrepancy in yaw and altitude.In some embodiments, additional points may be used to validate and/orcorrect for the solution obtained using two points.

Embodiments of the present invention include the system operating toconvert non-warped geographic mapping to spherical mapping. This processrequires multiple steps and conversion processes.

A first step is to convert the non-warped geographic map to the WGS84Geodedic coordinate system. The WGS84 Geodedic coordinate systemrepresents map positions as latitude, longitude and altitude. Anon-warped geographic map uses an image pyramid to represent a Mercatorprojection of the earth. The coarsest resolution image is in a 256 by256 square map image and it corresponds to a pyramid level zero or zoomlevel zero. Each successive zoom level doubles the size of thenon-warped geographic map image, so at level one the equator is 512pixels wide and at zoom level 17 the equator is more than 33.5 millionpixels wide.

The second step is to convert from the WGS84 Geodedic coordinate systemto the Earth-Centered-Earth-Fixed (“ECEF”) coordinate system. As statedpreviously, the WGS84 Geodetic coordinate system represents mappositions as latitude, longitude, and altitude. Since it is a geodeticcoordinate system it represents the altitude in meters measuredperpendicular to the surface of an oblique spheroid model of the Earthrather than measuring altitude from the center of the Earth. Of coursethe earth is not an exact spheroid and different approximations for theshape of the Earth have been proposed; however, the WGS84 standardrepresents a cross-section of the Earth as an ellipse with a minor axisaligned to the poles. The parameters of the ellipse are the length ofthe major axis A=6378137.0 meters and the length of the minor axisB=6356752.3142 meters.

The ECEF coordinate system is a Cartesian coordinate system with unitsas meters, and an origin at the center of the earth and axes alignedwith the North Pole, the prime meridian, and a third axis perpendicularto the first two. Because ECEF is a Cartesian system we can use it tomeasure distances and angles. The ECEF coordinates system is used as“World” coordinates. The following calculations may be performed by thesystem in order to convert from WGS84 to ECEF:

-   -   WGS84_A=6378137.0 # meters, radius of the earth    -   WGS84 E=8.1819190842622e-2 # eccentricity of the spheroid    -   def geo_to_ecef(geo_position):        -   lat, lon, alt=geo_position        -   a=WGS84_A        -   e=WGS84_E        -   # Calculation        -   lat=radians(lat)        -   lon=radians(lon)        -   n=a/sqrt(1.0−e**2*sin(lat)**2)        -   x=(n+alt)*cos(lat)*cos(lon)        -   y=(n+alt)*cos(lat)*sin(lon)        -   z=(n*(1−e**2)+alt)*sin(lat)        -   return x, y, z

The third step is to convert from ECEF coordinate system toEast-North-Up (“ENU”) local coordinate system. The ECEF coordinatesystem is earth-fixed and there is no clear notion of the local cardinaldirections (North, East, Up), which people on the surface of the Earthuse to orient ourselves. For example spherical images are likely to betaken so the East is zero degrees of pan, and North is 90 degrees ofpan, and West is 180 degrees of pan angle. However these are localdirections; depending on where you are in space, meaning that North maybe a different direction.

The ENU local coordinate system represents frame of reference with anarbitrary origin (e.g. the camera position) and with axes aligned to thelocal East, North, and Up directions. In the ENU system, the x-axis(East) is tangent to the gradient of ECEF with respect to longitude, they-axis is the gradient of ECEF with respect to latitude, and the z-axisis normal to the WGS84 ellipsoid. Again the units are in meters, and thereference system allows you to transform ECEF coordinates into localcoordinates relative to the cameras position. The following calculationsmay be performed by the system in order to convert from ECEF to ENU:

-   -   def create_enu_matrix(geopos):        -   lat, lon, alt=geopos        -   c0=cos(radians(lon)); s0=sin(radians(lon))        -   c1=cos(radians(lat)); s1=sin(radians(lat))        -   x, y, z=geo_to_ecef((lat, lon, alt))        -   # Set the locale ENU orientation matrix.        -   matrix=[[−s0, c0, 0, 0],            -   [−c0*s1, −s0*s1, c1, 0],            -   [c0*c1, s0*c1, s1, 0],            -   [0, 0, 0, 1]]        -   matrix=np.array(matrix, dtype=np.float64)        -   # Translate to the cameras position        -   matrix[:−1, 3]=−matrix.dot((x, y, z, 1.))[:−1]        -   return matrix

The fourth step is to convert the ENU local coordinate system to aspherical image to georectify the spherical image. The process toconvert a given geodetic location (lat, lon, alt) to a point on thespherical image is to first transform it to ECEF coordinates, and thenleft multiply the ENU matrix to generate points in a coordinate systemcentered at the camera. Then spherical coordinates of the point are:d=√((x^2±y^2±z^2))pan=(panOffset+180+[(tan)]^(−1)y/x)%360tilt=−[(tiltOffset+[(cos)]^(−1)z/d)%180]+90where panOffset and tiltOffset are corrections to the camera orientationthat can be set through the ASPHERE editor.

In a manual registration process, the system uses Thin Plate Spline(“TPS”) Warping. The actual warping is done to WGS84 geodeticcoordinates using a thin plate spline. A thin plate spline doesscattered data interpolation while minimizing the overall distortion(warping energy) of a two dimensional image. The warping happens whenthe first three points are selected/matched in the two images.

While it is shown and described to TPS, the system may use any otherspline or mathematical model to perform the functions of TPS. However,the other spline or mathematical model must have a way to create a trueor close enough inverse so forward and reverse warps can be executed.

Using TPS Warping, the system must first focus on a region of interest.Before selecting any points for warping, the system, operating theapplication displays a grid that is viewable on both images, as shown inFIG. 3, wherein the reference map 32 with grid 33 is displayed next tothe spherical image 34 with grid 35. The display of the grid does threethings.

First, it helps ensure accuracy within a certain distance. The distanceor span the grid encompasses is settable (ex: 2 miles from the center ofthe spherical image 34), as shown in FIG. 3. The center of the grid 35is the center of the spherical image 34. This guides the user to selectpoints within the grid 35 and this ensures the resulting warp will atleast be accurate within the region of interest (or grids 333 and 35).

Second, as points are picked and the warping algorithm takes effect, theuser can see the grid getting warped as well. This gives a visualfeedback to the user as to how much warp is needed for a particularimage. If the grid warps a lot, the user should pick many more points tostabilize the warp. Stabilizing the warp means adding more points willnot change the shape of the grid any more.

Third, at the initialization phase the grid acts as a guide to selectinitial altitude, pan, etc. values to align the images before thewarping process begins.

Each spherical image will carry this region of interest and thereforeany application on the other end that wishes to utilize the warp willhave some assurance as to the region of interest/accuracy for thatspherical image. This is useful in several situations. For example, andwithout limitation, a dense set of spherical images with overlappingcoverage. As the user is looking at one spherical image, he may want topick a point that is way outside of the region of interest. The systemmay operate to find out what the closest adjacent spherical image isw.r.t. the current spherical image and if that point falls in its regionof interest/accuracy and fetch the latitude, longitude and height valuesnot from the current spherical image but an adjacent spherical image.

The system includes a warper tool that allows a user to intake aspherical image 34 and a reference map 32, as shown in FIG. 3. Becausethe exact nature of errors is unknown, the system may supply the userwith tunable parameters such as altitude, tilt, pan and roll to rectifygenerally the orientation of the spherical image in reference to thereference map image. Latitude and Longitude can also be set. The usermay then start to click on a point 36 in reference map 32 and acorresponding point 38 appears in the spherical image 34. It isunderstood that the opposite can be done, wherein the user clicks on apoint 38 in spherical image 34 and a corresponding point 36 appears inthe reference map 32. The user then has the capability to move a point36 or 38 around to fix its location in either of the images. The points36 and 38 and their lat/lon appear in a column 40 to the left of theimages. These are called warp points 40 and become data used to generatethe TPS warp (forward). User skill set is needed to accurately representpoints and their true location in the reference map. After the firstthree points are rectified, the system may then begin to approximate theremainder of the points in the spherical image with reference to thereference map.

The system may also be used to blend the sky in order to produce thespherical image. The original spherical image shot by a camera systemmay capture part of the wings, wheels and other parts of the aircraftwhich occlude the sky portion of the spherical image as shown in FIG. 4a. The presence of these aircraft parts constitutes a visual disruption.The warper tool provides a way of blending an equirectangular image ofan unoccluded sky captured at a different time but visually similar tothe original spherical image, i.e., time of day, cloud patterns,atmospheric haze, etc.

This can be accomplished using two possible methods to composite/blendthe sky image onto the original spherical image. One method is a manualmethod and the other is an automated method.

With the manual method, the user is given control over roll, tilt, andpan to as closely align the image as possible visually. Based on theseinputs system operates the warper tool to warp the sky to match theorientation of the original spherical image as shown in FIG. 4b . Thenthe user is given control over the amount and location of blending ofthe sky image onto the original spherical image as shown in FIG. 4c . Inother words, the user can define the horizontal position (tilt angle)and the amount, in pixels, of fade (alpha channel dithering). The resultis a composite equirectangular image of the sky and original sphericalimage.

With the automated method, the warper tool would automatically analyzethe sky in the original spherical image shown in FIG. 4a to determinethe cloud patterns and atmospheric haze. The system would alsoautomatically read the time of day the original spherical image wascollected. The system would then automatically search a library of skiescollected from various times of day with various atmospheric conditions.The goal of the search is to find a sky in the library that most closelymatches the original spherical sky. Once the best match is chosen, thesystem would automatically read tilt, roll, heading values output froman inertial measurement unit at the time the original spherical imagewas captured and apply those values to the sky image to properly alignthe sky image with the original spherical image as shown in FIG. 4b .Finally, the warper tool would automatically apply a blend between thesky and original spherical images and output a composite equirectangularimage of the sky and original spherical image as shown in FIG. 4 c.

This new spherical image with the blended sky is used to create the cubemaps for ingesting by the deployment interface. Cube maps are maps onthe faces of a cube (six faces top, bottom, left, right, front andback). A spherical image can be represented by the six faces of a cube.Deployment interfaces use these as inputs to render the spherical imagefor the end user/viewer. The warper tool then generates the six cubemaps very fast.

In some embodiments, the deployment interface allows for reversing theprocess. The deployment interface may utilize an existing map API. Whenthe deployment interface starts a request is made to the API for mapdata. Next the deployment interface then checks to determine if thereare any spherical images.

The spherical images are then all added as places via the existing mapAPI. At this point a click event is linked to each point that opens upthe respective deployment interface. Before displaying the sphericalimage view, all of the currently plotted places on the existing map arescanned and converted from geodetic position to a panoramic positionutilizing the system as previously discussed. Once the panoramicposition has been obtained the system provides it to the deploymentinterface. The deployment interface may be configured by an XML filethat is also stored on the server and is also accessed via a singlerequest per spherical image. This configuration files stores deploymentinterface hotspots that are to be displayed in the panorama or sphericalview. To display places, objects and the like the system operates afterthe configuration file has been read and obtains the information, andmay use either a DOMParser or a new ActiveXObject. These interfaces thenallow the system to execute the read xml and edit any elements andattributes. The system then adds a hotspot element with pan and tiltattributes for each place. Finally, the deployment interface opens thespherical image ins a viewer and lets the viewer take control of theimage.

The deployment interface may include three components: a map viewer; apanorama viewer; and an interface which processes data retrieved frommultiple sources so that it is consistent with one another.

The map viewer is used to display top down data visualization in theexisting maps API. All the data displayed here trusts the trueness ofthe existing maps satellite imagery projections. This means that datasent to this interface doesn't require any preprocessing. The deploymentinterface can interface with any map service held on a map server,wherein the map server may be a third party map server.

The panorama viewer may be interfaced through JavaScript calls or thelike to add and/or remove data, adjust display parameters, etc. Oncedata has been prepared it is passed to the panorama viewer for display.An important aspect of embodiments of the present invention is that datacan be fed to the panoramic viewer.

Preparing data may be done on demand as requested by various featuresexposed to the user. The two types of requested data that need to bemade consistent, i.e. mapped in same geolocated projection, arespherical imagery and data markers. The geo projections need to beadjusted because there can be inconsistency in photographed sphericalimagery. To solve this issue, the data markers are converted to matchthe inconsistencies of the panoramic imagery. The reason the imageryisn't adjusted is that it would cause skewing that would be visible to afront end user.

The user may make requests to a centralized server, such as, but notlimited to sending restful API calls to the centralized server. Therequest contains a geolocation and a panorama identification number(“PID”). The server then returns a formatted object containing thegeolocation represented as a pan tilt pair representing the location ofthe corresponding data in the panorama associated with the PID. The pantilt calculated by the server uses the same library used to create thespherical image's associated warp. The system in performing thecalculation applies a thin plate spline transform to convert the actuallatitude and longitude to a warped projected latitude and longitude.Then a standard conversion from latitude, longitude, and altitude to panand tilt is applied.

The system can now utilize the deployment interface to initiate andperform “reverse” warping to get the (approximate) true location of apoint clicked in the panorama viewer. The reverse warp allows a user totake a point clicked in the spherical image and find its true geolocation as shown in FIG. 8, wherein points 54 and 56 clicked inspherical image 52 of deployment interface is then represented in itstrue deo location in the reference map 50 as points 53 and 55respectively. This is accomplished through what may be referred to as an“unwarp” or reverse warp process. Refer to hand drawn FIG. 2 on nextpage.

As shown in FIG. 5, for a point qi (x, y, z) located in the sphericalimage the system needs to find its true geolocation. The point C (cameraposition or center of the sphere) is a known point. The system creates avector V (C, qi) by connecting the center with qi. This point isrepresented in the ENU coordinate system. Next the system needs totransform this point qi into the ECEF system i.e. project it on “earth”.The system then uses the vector and projects or intersects it with theWGS84 ellipsoid representing earth. The projected point qi-hat has lat,lon and height (height=0 for 2D and Non Zero where DEM or 3D is used).This point represents the spherical image point (with error) projectedon the earth's surface (ECEF). The system now needs to “reverse warp” toget the point's true location in the sphere (lat, lon, h).

If Wf represents the forward warp and is constructed from points Pi andGi (Wf=tps [Pi, Gi]) as described previously, then the system canconstruct another warp called Wr=tps[Gi, Pi]. While Wr is not a trueinverse of Wf, it is a very close approximation as its inverse. Thesystem may then apply Wr to point qi-hat to get point pi-hat on thespherical image.

In some embodiments, correcting relief displacement in equirectangularimages is necessary. Relief displacement is a result of perspectiveprojection in aerial images which causes elevated objects to appear tolean away from the focal point of the camera. This applies to largefeatures such as hills, mountains, and valleys as well as to fine scalefeatures such as rooftops. In remote sensing applications in whichimages are the results of optics (rather than the complex warpingprocess used to produce panoramic images). As shown in FIG. 6, depictedis an example of the Shuttle Radar Topography MIssion (SRTM) 1 ArcSecond Global DEM data over the Phoenix metropolitan area. This data hasa spatial resolution of 30 meters per pixel; brighter colors indicatehigher elevation measured geodetically (normal to the surface of theWGS84 spheroid rather than from its center).

With reference to FIGS. 7a-7d , embodiments of the present inventioninclude an iterative process for correcting relief displacement byestimating the intersection of a ray and the displaced surface of theEarth. In FIG. 7a , (1) A ray is cast from the camera's center (justbelow the aircraft) out through an imaginary sphere until it intersectsthe WGS84 spheroid; in FIG. 7b , (2) The elevation at the first point ofintersection is fetched from the DEM and a new intersection point iscalculate between the ray and a dilated version of the spheroid that isdilated to match the DEM elevation, so that the latitude and longitudecoordinates of the second point can be determined by projecting the newintersecting point onto the WGS ellipsoid and fetching the elevation atthe new location from a DEM; in FIG. 7c , (3) The process repeats usinga spheroid that is dilated to the elevation of the second point; in FIG.7d , (4) at some point the difference in elevation between the lastintersection point and the DEM data falls below a tolerance threshold,or in some embodiments, until the system exceeds a maximum number ofallowed iterations.

Another approach, according to embodiments of the present inventionwould be to do a bisection search along the ray to find the intersectionthrough operation of the system. First, calculate d_max=intersectionwith the WGS84 ellipsoid. Then fetch the elevation from a DEM andintersect it with the ray. If the elevation at the new point is abovethe identified point, update d_max and repeat the process. Eventually,the system will identify a point above the ray and can set the d_min tothe distance along this ray. The actual intersection between the surfaceand the ray must lie between d_min and d_max, so we can probe the ray at(d_min+d_max)/2. If the elevation at this middle point is under the ray,the system updates d_max:=(d_min+d_max)/2 and repeats. If this is belowthe ray, the system updates d_min and repeats. This guaranties that thesystem cuts the error in half at each step, and will converge quickly.

In fact, in some embodiments, the entire process may be done this way;set d_max to the intersection with a WGS spheroid model of the earth,set d_min to zero, and do a bisection search for the intersection withthe DEM.

The embodiments and examples set forth herein were presented in order tobest explain the present invention and its practical application and tothereby enable those of ordinary skill in the art to make and use theinvention. However, those of ordinary skill in the art will recognizethat the foregoing description and examples have been presented for thepurposes of illustration and example only. The description as set forthis not intended to be exhaustive or to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the teachings above without departing from the spirit andscope of the forthcoming claims.

The invention claimed is:
 1. A post capture imagery processing systemfor use with aerial spherical imagery, the system comprising: a servercomprising a memory, a database stored in the memory, and a processor,wherein the database contains data for a spherical image and data for areference map; and a computer coupled to the server wherein a userinterfaces with the server through the computer, wherein the server isprogrammed to: automatically create a non-linear warp from a referencegeodetic coordinate system to a pan and tilt coordinates in anequirectangular image; display the equirectangular image on a display ofthe computer; automatically retrieving a georectified location of apoint in the equirectangular image in response to receiving a signalfrom the computer; automatically create the nonlinear warp, the serveris further programmed to convert a non-warped geographic map to a WGS84Geodedic coordinate system; convert from the WGS84 Geodedic coordinatesystem to Earth-Centered-Earth-Fixed coordinate system; convert from theEarth-Centered-Earth-Fixed coordinate system to an East-North-Up localcoordinate system; and convert from the East-North-Up local coordinatesystem to a spherical image, wherein the conversion georectifies thespherical image.
 2. The system of claim 1, wherein to automaticallycreate the non-linear warp the server is further programmed toautomatically perform a registration process.
 3. The system of claim 2,wherein the registration process is a manual registration process usinga thin plate spline warping process, wherein the system provides a userinterface that interactively prompts and displays information to theuser to guide the user through the manual registration process.
 4. Thesystem of claim 2, wherein the registration process is an automatedregistration process.
 5. A method of using a post capture imageryprocessing system to georectify an aerial spherical image, the methodcomprising: storing data for a spherical image and data for a referencemap in a database of a server comprising a memory and a processor,wherein the database is stored in the memory; coupling the server to acomputer, wherein a user interfaces with the server through thecomputer; automatically creating at the server a non-linear warp from areference geodetic coordinate system to a pan and tilt coordinates in anequirectangular image; display at the computer from the server theequirectangular image; automatically retrieving a georectified locationof a point in the equirectangular image by the server in response toreceiving a signal from the computer; wherein automatically creating thenon-linear warp further comprises converting a non-warped geographic mapto a WGS84 Geodedic coordinate system at the server; converting from theWGS84 Geodedic coordinate system to Earth-Centered-Earth-Fixedcoordinate system at the server; converting from theEarth-Centered-Earth-Fixed coordinate system to an East-North-Up localcoordinate system at the server; and converting from the East-North-Uplocal coordinate system to a spherical image at the server, wherein theconversion georectifies the spherical image.
 6. The method of claim 5,further comprising performing a registration process at the server. 7.The method of claim 6, wherein performing the registration processcomprises performing a manual registration process using a thin platespline warping process, wherein the system provides a user interfacethat interactively prompts and displays information to the user to guidethe user through the manual registration process.
 8. The method of claim6, wherein performing the registration process comprises performing anautomated registration process.